perf: Reduce record batch memory accounting overhead - #24319
Open
ryux1 wants to merge 3 commits into
Open
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #24319 +/- ##
==========================================
+ Coverage 81.13% 81.18% +0.04%
==========================================
Files 1112 1109 -3
Lines 388220 388301 +81
Branches 388220 388301 +81
==========================================
+ Hits 314985 315240 +255
+ Misses 54696 54516 -180
- Partials 18539 18545 +6 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Extend the microbenchmark across row counts and List/Struct layouts. Exercise all legal run-end index types in the ArrayData parity test.
ryux1
force-pushed
the
perf/batch-memory-accounting
branch
from
August 13, 2026 18:14
e584d8b to
890be61
Compare
Author
|
Thanks @comphead — added in 890be61:
The updated A/B tables are in the PR description. The row-count cases stay essentially flat as expected for buffer traversal; measured speedups are 7.49–7.97x for four primitive columns, 9.39x for lists, and 8.86x for structs.
|
Contributor
|
Thanks @ryux1 -- I'll check it out shortly |
Contributor
|
run benchmark window_query_sql |
1 similar comment
Contributor
|
run benchmark window_query_sql |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/batch-memory-accounting (890be61) to 8c099dc (merge-base) diff Run configurationrun benchmark window_query_sqlResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/batch-memory-accounting (890be61) to 8c099dc (merge-base) diff Run configurationrun benchmark window_query_sqlResults will be posted here when complete File an issue against this benchmark runner |
comphead
reviewed
Aug 13, 2026
Clarify that the 16-entry threshold avoids allocations for small buffer sets while bounding inline storage and linear lookup. The threshold is a performance heuristic rather than a semantic limit.
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing perf/batch-memory-accounting (890be61) to 8c099dc (merge-base) diff Run configurationrun benchmark window_query_sqlCPU Details (lscpu)Details
Resource Usagewindow_query_sql — base (merge-base)
window_query_sql — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing perf/batch-memory-accounting (890be61) to 8c099dc (merge-base) diff Run configurationrun benchmark window_query_sqlCPU Details (lscpu)Details
Resource Usagewindow_query_sql — base (merge-base)
window_query_sql — branch
File an issue against this benchmark runner |
Contributor
|
Looks like a small but real improvement to me -- thank you @ryux1 |
Contributor
|
run benchmarks |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/batch-memory-accounting (db959c3) to 8c099dc (merge-base) diff Run configurationrun benchmark tpcdsResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/batch-memory-accounting (db959c3) to 8c099dc (merge-base) diff Run configurationrun benchmark clickbench_partitionedResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/batch-memory-accounting (db959c3) to 8c099dc (merge-base) diff Run configurationrun benchmark tpchResults will be posted here when complete File an issue against this benchmark runner |
alamb
reviewed
Aug 14, 2026
| group.finish(); | ||
| } | ||
|
|
||
| criterion_group!( |
Contributor
There was a problem hiding this comment.
If we want to keep this benchmark, can you please add some comments explaining what it is testing (specifically the overhead of measuring memory usage in record batches)?
| fn count_array_data_memory_size( | ||
| array_data: &ArrayData, | ||
| counted_buffers: &mut HashSet<NonZero<usize>>, | ||
| /// Tracks a small number of buffers inline, avoiding a heap allocation for |
| } | ||
| } | ||
|
|
||
| fn count_buffer_memory_size( |
Contributor
There was a problem hiding this comment.
This looks like it could be a method on RecordBatchMemoryCounter as it is taking two mut fields from it and updating them (I think it is logically part of that logic). Not required, just a suggestion
| total_size, | ||
| ); | ||
| }, | ||
| _ => unreachable!(), |
Contributor
There was a problem hiding this comment.
why is this unreachable? An unreachable results in a panic at runtime
I suggest:
- explicitly list out all array types that are not uspported (rather than
_catch all) so it is explict what is not being handled - (probably) implement support for the relevant types (or leave a TODO comment referencing a ticket to support them)
|
|
||
| fn count_byte_array_memory_size<T: ByteArrayType>( | ||
| array: &arrow::array::GenericByteArray<T>, | ||
| counted_buffers: &mut BufferIdSet, |
Contributor
There was a problem hiding this comment.
similarly I think this could be a method on RecordBatchMemoryCounter
| count_buffer_memory_size(array.values(), counted_buffers, total_size); | ||
| } | ||
|
|
||
| fn count_list_array_memory_size<O: arrow::array::OffsetSizeTrait>( |
Contributor
There was a problem hiding this comment.
list view and byte view arrays also probably need special handling
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing perf/batch-memory-accounting (db959c3) to 8c099dc (merge-base) diff Run configurationrun benchmark tpchCPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing perf/batch-memory-accounting (db959c3) to 8c099dc (merge-base) diff Run configurationrun benchmark tpcdsCPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing perf/batch-memory-accounting (db959c3) to 8c099dc (merge-base) diff Run configurationrun benchmark clickbench_partitionedCPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
comphead
reviewed
Aug 14, 2026
comphead
left a comment
comphead
left a comment
Contributor
There was a problem hiding this comment.
We may want to consider adding a bench that reuses a RecordBatchMemoryCounter across N sliced batches (the shared-buffer case), which is the hash-join build-side workload at datafusion/physical-plan/src/joins/hash_join/exec.rs:2134. That's where the inline-16 fast path matters most.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
Record batch memory accounting runs on hot execution paths. The current implementation materializes ArrayData for every array and allocates a hash set even for small batches, adding measurable overhead to queries that frequently update memory reservations.
What changes are included in this PR?
Criterion point estimates from
cargo bench -p datafusion-common --bench record_batch_memory, measured sequentially on upstreammainand this PR on the same machine:Column count (8,192 rows, Int64)
Row count (4 Int64 columns)
Array layout (4 columns, 8,192 rows)
Each list row contains two Int64 values. Each struct column contains two Int64 child fields.
Are these changes tested?
Yes.
cargo fmt --all -- --checkcargo clippy --all-targets --all-features -- -D warningscargo test -p datafusion-common utils::memory --lib(10 passed)RUST_BACKTRACE=1 cargo test --profile ci --exclude datafusion-examples --exclude datafusion-benchmarks --exclude datafusion-cli --workspace --lib --tests --bins --features avro,json,backtrace,extended_tests,recursive_protection,parquet_encryptionAre there any user-facing changes?
No API or behavior changes. This reduces CPU and allocation overhead in record batch memory accounting.
AI assistance: OpenAI Codex assisted with implementation and test execution. I reviewed the change and its behavior end to end.